Papers with STS datasets
Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings have not been well understood what properties are captured in the resulting sentences depending on the supervision signals. |
| Approach: | They propose to combine two types of sentence embedding methods with similar architectures and tasks to investigate their properties. |
| Outcome: | The proposed methods perform better on unsupervised and downstream tasks than the proposed methods on untrained STS tasks and probing tasks. |
MUSTS: MUltilingual Semantic Textual Similarity Benchmark (2025.acl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for semantic textual similarity (STS) are limited to high-resource languages and do not include datasets annotated focusing on relatedness instead of similarity. |
| Approach: | They propose to evaluate multilingual semantic textual similarity benchmarks which span 13 languages and annotated datasets to evaluate and compare them. |
| Outcome: | The proposed method is the most comprehensive benchmark of multilingual STS methods. |
KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmark datasets for natural language inference and semantic textual similarity (STS) are not available in the Korean language. |
| Approach: | They construct and release new datasets for Korean NLI and STS . they machine-translate existing English training sets and manually translate development and test sets into Korean to accelerate research on Korean NLU. |
| Outcome: | The proposed datasets are available at https://github.com/kakaobrain/KorNLUDatasets. |
Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity (2023.acl-short)
Copied to clipboard
| Challenge: | Semantic Textual Similarity (STS) measures the degree to which the underlying semantics of paired sentences are equivalent. |
| Approach: | They propose a token-level matching inference algorithm which can be applied on top of any language model to improve its performance on STS task. |
| Outcome: | The proposed method improves the performance of almost all language models, with up to 12.7% gain in Spearman’s correlation. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
Fine-grained Semantic Textual Similarity for Serbian (L18-1)
Copied to clipboard
| Challenge: | Semantic textual similarity (STS) is a task of assigning a numerical score to short texts based on the level of semantic equivalence between them. |
| Approach: | They propose to annotate Serbian STS dataset with fine-grained similarity scores . they propose a supervised bag-of-words model that combines part-of speech weighting with term frequency weighting . |
| Outcome: | The proposed model outperforms existing models on the Serbian STS News Corpus . the proposed model is based on a new morphologically rich language . |
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing BERT-based pre-trained language models achieve high performance on many downstream tasks, but native derived sentence representations are collapsed and thus poor performance on semantic textual similarity (STS) tasks. |
| Approach: | They propose a framework for self-supervised Sentence Representation Transfer that adopts contrastive learning to fine-tune BERT in an unsupervised way. |
| Outcome: | The proposed framework improves on the BERT-derived representations by 8% on STS datasets and shows robustness in data scarcity scenarios. |
Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing supervised sentence embedding techniques rely on expensive human-annotated sentence pairs as the supervised signals. |
| Approach: | They propose a semi-supervised sentence embedding framework that leverages large-scale unlabeled data. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on four domain adaptation tasks. |
EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to measure semantic similarity between biomedical texts are inefficient due to too many biomedically-related entities. |
| Approach: | They propose an entity-aligned, attention-based and retrieval-augmented PLM that aligns the same type of fine-grained entity information in each sentence pair with an entity alignment matrix with an auxiliary loss. |
| Outcome: | The proposed model can achieve state-of-the-art on both in-domain and out-of domain datasets. |